Measuring the Effect of Code Complexity on Static Analysis Results

To understand the effect of code complexity on static analysis, thirty-five format string vulnerabilities were selected from the National Vulnerability Database. We analyzed two sets of code for each vulnerability. The first set of code contained the vulnerability, while the second was a later version of the code in which the vulnerability had been fixed. We examined the effect of both code complexity and the year of discovery on the quality of static analysis results, including successful detection and false positive rates. The tool detected 63% of the format string vulnerabilities, with detection rates decreasing with increasing code complexity. When the tool failed to detect a bug, it was for one of two reasons: the absence of security rules specifying the vulnerable function, or the presence of a bug in the static analysis tool. Complex code is more likely to contain complicated code constructs and obscure format string functions, resulting in lower detection rates. However, detection rates did not change substantially from 2000 to 2006, showing that reported format string vulnerabilities are not becoming more difficult to